Activity detection in web applications

ABSTRACT

A computing system includes a web server, client computing devices, a proxy between the web server and the client computing devices, and an analytics server. Each client computing device is operated by an end-user to access an application based on end-user events resulting in representational state transfer (REST) calls to the web server. The proxy passes through the REST calls to the web server and returns responses from the web server, with the return responses corresponding to activities being performed within the web application. The analytics server correlates the end-user events with the corresponding REST calls and return responses from the proxy for each client computing device, and uses vectorization to compare similar activities. The analytics server associates the similar activities with a quality indicator to identify anomalies within the application for corrective action to be taken.

TECHNICAL FIELD

The present disclosure relates to web applications, and moreparticularly, to analyzing activities performed within an applicationacross a large set of end-users.

BACKGROUND

Software as a Service (SaaS) is a computing approach to softwaredelivery by which applications are centrally hosted on one or morethird-party servers (e.g., in the cloud) and are typically provided on asubscription basis to users. SaaS applications are offered for numerousareas, including business, accounting, and even gaming applications.

SaaS applications are typically accessed on client computing devices viaan internet browser. A large set of end-users within an enterprise mayaccess the same SaaS application, where each end-user causes the SaaSapplication to perform various activities.

Many of the activities performed within the application are similaracross the large set of end-users. Since SaaS applications are onthird-party servers, it is difficult for an enterprise to monitoractivities of their end-users within a SaaS application since theenterprise does not own the server providing the SaaS application.

SUMMARY

A computing system includes a web server to provide an application andclient computing devices. Each client computing device is operated by anend-user to access the application based on end-user events resulting inrepresentational state transfer (REST) calls to the web server. A proxyis between the web server and the client computing devices, and isconfigured to pass through the REST calls to the web server and toreturn responses from the web server, with the return responsescorresponding to activities being performed within the application.

An analytics server is configured to receive the end-user events fromeach client computing device, and to receive the REST calls and thereturn responses from the proxy. The analytics server comprises aprocessor configured to correlate the end-user events with thecorresponding REST calls and return responses from the proxy for eachclient computing device, and translate respective correlated end-userevents, REST calls and return responses into respective event vectors.The analytics server processes the respective event vectors to determinesimilarities among the client computing devices, with the similaritiescorresponding to similar activities, and associates the similaractivities with a quality indicator to identify anomalies within theapplication for corrective action to be taken.

The quality indicator may be based on one or more statistics associatedwith the end-users accessing the application. The corrective action maycomprise recommending to one or more end-users which end-users events touse for more efficiently performing a particular activity within theapplication.

Processing the respective event vectors by the processor may compriseplacing the respective event vectors in sequence for each clientcomputing device, and selecting an n-set of event vectors in sequencefrom the sequence of event vectors for each client computing device,where n>1. An activity within the application corresponds to theselected n-set of event vectors in sequence. A composite activity vectoris generated by adding together the event vectors in the n-set of eventvectors in sequence for each client computing device. The compositeactivity vectors for each client computing device are placed in vectorspace. The similarities among the client computing devices aredetermined based on similarities between the composite activity vectorsamong the client computing devices.

Each client computing device comprises an embedded browser to monitorthe end-user events by capturing events at the document object model(DOM) level within a web page.

Each client computing device comprises a processor configured tocooperate with the embedded browser to extract end-user event attributesfrom the DOM, with the end-user event attributes providing contextualinformation on the corresponding end-user event. The analytics serverreceives the contextual information associated with the end-user events,and is further configured to include the contextual information in therespective event vectors.

The corresponding REST calls and return responses from the proxy includeparameter information noise, and where the analytics server is furtherconfigured to remove the parameter information noise by applying one ormore reducer functions. Applying the one or more reducer functionscomprises translating each event vector into one or more reduced eventvectors, with the resulting reduced event vectors being used as theevent vectors by the analytics server.

The n-set of event vectors in sequence corresponds to a pattern thatincludes a beginning and an end of the corresponding activity. Theanalytics server is further configured to incrementally vary n for then-set of event vectors, generate an activity vector for each value of n,examine each activity vector for a pattern, and select the ncorresponding to the activity vector having a pattern.

Another aspect is directed to a method for operating an analytics serverwithin a computing system as described above, with the method comprisingreceiving the end-user events from each client computing device,receiving the REST calls and the return responses from the proxy,correlating the end-user events with the corresponding REST calls andreturn responses from the proxy for each client computing device, andtranslating respective correlated end-user events, REST calls and returnresponses into respective event vectors. The method further includesprocessing the respective event vectors to determine similarities amongthe client computing devices, with the similarities corresponding tosimilar activities, and associating the similar activities with aquality indicator to identify anomalies within the application forcorrective action to be taken.

Yet another aspect is directed to a non-transitory computer readablemedium for operating an analytics server as described above, and withthe non-transitory computer readable medium having a plurality ofcomputer executable instructions for causing the analytics server toperform steps as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a network environment of computing devicesin which various aspects of the disclosure may be implemented.

FIG. 2 is a block diagram of a computing device useful for practicing anembodiment of the client machines or the remote machines illustrated inFIG. 1.

FIG. 3 is a block diagram of a computing system with an analytics serverfor analyzing similar activities performed within an application acrossa large set of end-users in which various aspects of the disclosure maybe implemented.

FIG. 4 is a general flowchart illustrating a method for operating thecomputing system illustrated in FIG. 3.

FIG. 5 is a general flowchart illustrating a method for operating theanalytics server illustrated in FIG. 3.

FIG. 6 is a more detailed flowchart for processing the respective eventvectors in the flowchart of FIG. 5.

DETAILED DESCRIPTION

The present description is made with reference to the accompanyingdrawings, in which exemplary embodiments are shown. However, manydifferent embodiments may be used, and thus the description should notbe construed as limited to the particular embodiments set forth herein.Rather, these embodiments are provided so that this disclosure will bethorough and complete. Like numbers refer to like elements throughout.

As will be discussed below, a computing system with an analytics serveris used for analyzing similar activities performed within an applicationacross a large set of end-users. Vectorization techniques are used tocompare similar activities being performed within the application acrossthe large set of users. The similar activities are associated with aquality indicator to identify anomalies within the application forcorrective action to be taken.

As will be appreciated by one of skill in the art upon reading thefollowing disclosure, various aspects described herein may be embodiedas a device, a method or a computer program product (e.g., anon-transitory computer-readable medium having computer executableinstruction for performing the noted operations or steps). Accordingly,those aspects may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment combining software andhardware aspects.

Furthermore, such aspects may take the form of a computer programproduct stored by one or more computer-readable storage media havingcomputer-readable program code, or instructions, embodied in or on thestorage media. Any suitable computer readable storage media may beutilized, including hard disks, CD-ROMs, optical storage devices,magnetic storage devices, solid-state storage devices, and/or anycombination thereof.

Referring initially to FIG. 1, a non-limiting network environment 101 inwhich various aspects of the disclosure may be implemented includes oneor more client machines 102A-102N, one or more remote machines106A-106N, one or more networks 104, 104′, and one or more appliances108 installed within the computing environment 101. The client machines102A-102N communicate with the remote machines 106A-106N via thenetworks 104, 104′.

In some embodiments, the client machines 102A-102N communicate with theremote machines 106A-106N via an intermediary appliance 108. Theillustrated appliance 108 is positioned between the networks 104, 104′and may be referred to as a network interface or gateway. In someembodiments, the appliance 108 may operate as an application deliverycontroller (ADC) to provide clients with access to business applicationsand other data deployed in a datacenter, the cloud, or delivered asSoftware as a Service (SaaS) across a range of client devices, and/orprovide other functionality such as load balancing, etc. In someembodiments, multiple appliances 108 may be used, and the appliance(s)108 may be deployed as part of the network 104 and/or 104′.

The client machines 102A-102N may be generally referred to as clientmachines 102, local machines 102, clients 102, client nodes 102, clientcomputers 102, client devices 102, computing devices 102, endpoints 102,or endpoint nodes 102. The remote machines 106A-106N may be generallyreferred to as servers 106 or a server farm 106. In some embodiments, aclient device 102 may have the capacity to function as both a clientnode seeking access to resources provided by a server 106 and as aserver 106 providing access to hosted resources for other client devices102A-102N. The networks 104, 104′ may be generally referred to as anetwork 104. The networks 104 may be configured in any combination ofwired and wireless networks.

A server 106 may be any server type such as, for example: a file server;an application server; a web server; a proxy server; an appliance; anetwork appliance; a gateway; an application gateway; a gateway server;a virtualization server; a deployment server; a Secure Sockets Layer orTransport Layer Security Virtual Private Network (SSL/TLS VPN) server; afirewall; a web server; a server executing an active directory; or aserver executing an application acceleration program that providesfirewall functionality, application functionality, or load balancingfunctionality.

A server 106 may execute, operate or otherwise provide an applicationthat may be any one of the following: software; a program; executableinstructions; a virtual machine; a hypervisor; a web browser; aweb-based client; a client-server application; a thin-client computingclient; an ActiveX control; a Java applet; software related to voiceover Internet protocol (VoIP) communications like a soft IP telephone;an application for streaming video and/or audio; an application forfacilitating real-time-data communications; a HTTP client; a FTP client;an Oscar client; a Telnet client; or any other set of executableinstructions.

In some embodiments, a server 106 may execute a remote presentationclient or other client or program that uses a thin-client or aremote-display protocol to capture display output generated by anapplication executing on a server 106 and transmits the applicationdisplay output to a client device 102.

In yet other embodiments, a server 106 may execute a virtual machineproviding, to a user of a client device 102, access to a computingenvironment. The client device 102 may be a virtual machine. The virtualmachine may be managed by, for example, a hypervisor, a virtual machinemanager (VMM), or any other hardware virtualization technique within theserver 106.

In some embodiments, the network 104 may be: a local-area network (LAN);a metropolitan area network (MAN); a wide area network (WAN); a primarypublic network 104; and a primary private network 104. Additionalembodiments may include a network 104 of mobile telephone networks thatuse various protocols to communicate among mobile devices. For shortrange communications within a WLAN, the protocols may include 802.11,Bluetooth, and Near Field Communication (NFC).

FIG. 2 depicts a block diagram of a computing device 100 useful forpracticing an embodiment of client devices 102 or servers 106. Thecomputing device 100 includes one or more processors 103, volatilememory 122 (e.g., random access memory (RAM)), non-volatile memory 128,user interface (UI) 123, one or more communications interfaces 118, anda communications bus 150.

The non-volatile memory 128 may include: one or more hard disk drives(HDDs) or other magnetic or optical storage media; one or more solidstate drives (SSDs), such as a flash drive or other solid state storagemedia; one or more hybrid magnetic and solid state drives; and/or one ormore virtual storage volumes, such as a cloud storage, or a combinationof such physical storage volumes and virtual storage volumes or arraysthereof.

The user interface 123 may include a graphical user interface (GUI) 124(e.g., a touchscreen, a display, etc.) and one or more input/output(I/O) devices 126 (e.g., a mouse, a keyboard, a microphone, one or morespeakers, one or more cameras, one or more biometric scanners, one ormore environmental sensors, and one or more accelerometers, etc.).

The non-volatile memory 128 stores an operating system 115, one or moreapplications 116, and data 117 such that, for example, computerinstructions of the operating system 115 and/or the applications 116 areexecuted by processor(s) 103 out of the volatile memory 122. In someembodiments, the volatile memory 122 may include one or more types ofRAM and/or a cache memory that may offer a faster response time than amain memory. Data may be entered using an input device of the GUI 124 orreceived from the I/O device(s) 126. Various elements of the computer100 may communicate via the communications bus 150.

The illustrated computing device 100 is shown merely as an exampleclient device or server, and may be implemented by any computing orprocessing environment with any type of machine or set of machines thatmay have suitable hardware and/or software capable of operating asdescribed herein.

The processor(s) 103 may be implemented by one or more programmableprocessors to execute one or more executable instructions, such as acomputer program, to perform the functions of the system. As usedherein, the term “processor” describes circuitry that performs afunction, an operation, or a sequence of operations. The function,operation, or sequence of operations may be hard coded into thecircuitry or soft coded by way of instructions held in a memory deviceand executed by the circuitry. A processor may perform the function,operation, or sequence of operations using digital values and/or usinganalog signals.

In some embodiments, the processor can be embodied in one or moreapplication specific integrated circuits (ASICs), microprocessors,digital signal processors (DSPs), graphics processing units (GPUs),microcontrollers, field programmable gate arrays (FPGAs), programmablelogic arrays (PLAs), multi-core processors, or general-purpose computerswith associated memory.

The processor may be analog, digital or mixed-signal. In someembodiments, the processor may be one or more physical processors, orone or more virtual (e.g., remotely located or cloud) processors. Aprocessor including multiple processor cores and/or multiple processorsmay provide functionality for parallel, simultaneous execution ofinstructions or for parallel, simultaneous execution of one instructionon more than one piece of data.

The communications interfaces 118 may include one or more interfaces toenable the computing device 100 to access a computer network such as aLocal Area Network (LAN), a Wide Area Network (WAN), a Personal AreaNetwork (PAN), or the Internet through a variety of wired and/orwireless connections, including cellular connections.

In described embodiments, the computing device 100 may execute anapplication on behalf of a user of a client device. For example, thecomputing device 100 may execute one or more virtual machines managed bya hypervisor. Each virtual machine may provide an execution sessionwithin which applications execute on behalf of a user or a clientdevice, such as a hosted desktop session. The computing device 100 mayalso execute a terminal services session to provide a hosted desktopenvironment. The computing device 100 may provide access to a remotecomputing environment including one or more applications, one or moredesktop applications, and one or more desktop sessions in which one ormore applications may execute.

Additional descriptions of a computing device 100 configured as a clientdevice 102 or as a server 106, or as an appliance intermediary to aclient device 102 and a server 106, and operations thereof, may be foundin U.S. Pat. Nos. 9,176,744 and 9,538,345, which are incorporated hereinby reference in their entirety. The '744 and '345 patents are bothassigned to the current assignee of the present disclosure.

Referring initially to FIG. 3, the illustrated computing system 10includes client computing devices 20(1)-20(n), a web server 40 providingan application 42, a proxy 50 and an analytics server 60. The proxy 50is positioned between the client computing devices 20(1)-20(n) and theweb server 40. The analytics server 60 is in communications with theclient computing devices 20(1)-20(n) and the proxy 50, and analyzesactivities within the web application 42 as initiated by end-users ofthe client computing devices 20(1)-20(n).

As will be discussed in detail below, the analytics server 60 analyzessequences of representational state transfer (REST) calls related toend-user tasks. Similar activities among the end-users are detectedusing vectorization techniques, with the similar activities beingassociated with quality indicators to identify anomalies across theend-users.

The quality indicator may be based on one or more statistics associatedwith the end-users accessing the application 42. Corrective action maybe directed to quality improvement recommendations, for example, forend-users that are not as efficient in performing similar activitieswithin the application 42 as compared to other end-users. As such, thesystems and methods set forth herein advantageously provide improvedperformance within a virtualized and/or enterprise computingenvironment.

The client computing devices 20(1)-20(n) are generally referred to asclient computing devices 20 and typically operate within an enterprise.Each of the end-users of the client computing devices 20(1)-20(n) accessthe same application 42 within the web server 40.

The number of computing devices 20 may vary from several hundred toseveral thousand, for example. The statistical data tends to be moreaccurate for larger sets of client computing devices 20. Statisticaldata may include calculation of standard deviations for the end-usersoperating the client computing devices 20 for performing particularactivities within the application 42. Statistical data may also includecalculation on the frequency and speed of performing particularactivities within the application 42.

Each client computing device 20 includes an embedded browser 24 toaccess the application 42. The embedded browser 24 is a browser embeddedwithin a native application 22. For example, Citrix Receiver and CitrixWorkspace App are programs that are installed on client computingdevices 20.

Since the embedded browser 24 is embedded within the native application22 within the client computing device 20, this allows end-user events tobe collected. End-user events include mouse clicks and key strokes, forexample. One or more input devices 26 coupled to the native application22 are used to generate the end-user events. A mouse and keyboard areexample input devices 26.

More particularly, the end-user events are collected at the documentobject model (DOM) level within a web page. This advantageously allowsattributes associated with the end-user events to be extracted from theDOM. Example attributes include the label of a button that has beenclicked, page titles and the likes. Attributes may be used to providecontextual information corresponding to the activities being performedwithin the application 42. Each client computing device 20 providesend-user events and their corresponding attributes to the analyticsserver 60.

As readily appreciated by those skilled in the art, most applicationsare driven by representational state transfer (REST) services, whereeach end-user request results in one or more REST calls to the webserver 40. Activities are performed within the application 42 inresponse to the received REST calls.

REST is a web design model used by many cloud service providers,enterprises, and social media companies to define interfaces with theirservices. A cloud service provider may offer scalable computingresources as services over networks, such as provisioning virtualmachine instances which can run enterprise applications for customers.Many cloud storage service providers use REST to define data integrationAPIs that may be invoked to extract and load data, among other things.

Web services that conform to the REST architectural style, termedRESTful web services, provide interoperability between computer systemson the Internet. RESTful web services allow the requesting systems toaccess and manipulate textual representations of web resources by usinga uniform and predefined set of stateless operations.

The web server 40 providing the application 42 is typically not owned bythe enterprise. This is particularly so when the web server 40 is aSoftware as a Service (SaaS) server, and the web application 42 is aSaaS application. Consequently, the enterprise does not have access tothe web server 40 for installing equipment to monitor activities withinthe application 42 across each of the client computing devices 20.

Instead, the illustrated computing system 10 includes a proxy 50 forlistening to server requests and responses from the server 40. Itappears that the client computing devices 20 are communicating directlywith the web server 40, but the client computing devices 20 are actuallycommunicating with the proxy 50.

The proxy 50 is forwarding the REST calls to the web server 40 and thencollects the results to send back to the client computing devices 20.Since the proxy 50 is acting as a man-in-the-middle, the proxy 50 isable to forward the REST calls and the return responses to the analyticsserver 60.

As will now be discussed in greater detail, the analytics server 60 isconfigured to analyze sequences of REST calls related to end-user tasksfor each client computing device 20. For discussion purposes, theillustrated application 42 is SalesForce. SalesForce is a customerrelationship management (CRM) application provided by SalesForce.com,Inc.

An example activity within the application 42 is when an end-user goesfrom a log-in page to a particular customer in SalesForce. This activityis based on a series of end-users events. The end-user events may bedifferent for different end-users. For example, end-users may use adifferent number of key strokes or mouse clicks to go from the log-inpage to a particular customer. As an example, to get to a particularcustomer one approach involves using a single mouse click and anotherapproach is to use a couple of keystrokes, such as command C to go to acustomer.

Most applications, such as SalesForce, have customization levels anddata submitted in each REST call that is likely to be different for eachrequest. Detecting similar sequences related to an activity cannot beperformed on simple compares of REST calls. It is also difficult todetect the beginning and end of an activity without control over theend-user environment (i.e., the web server 40).

The above limitations are overcome by the analytics server 60communicating with the client computing devices 20 and with the proxy 50via the network 104. The analytics server 60 includes a processor 62 anda memory 64. The memory 64 stores end-user events and associatedcontextual information as received from the client computing devices 20,and stores REST calls and return responses as received from the proxy50.

The processor 62 correlates the end-user events with the correspondingREST calls and return responses from the proxy for each client computingdevice 20. The correlations also include the end-user's intent for eachend-user event.

As noted above, the embedded browser 22 within each client computingdevice 20 monitors the end-user events by capturing events at the DOMlevel within a web page. The processor 28 within each client computingdevice 20 is configured to cooperate with the embedded browser 22 toextract end-user event attributes from the DOM, with the end-user eventattributes providing contextual information on the correspondingend-user event, i.e., the end-user's intent. Example attributes includethe label of a button that has been clicked, page titles and the likes.

The processor 62 within the analytics server 60 translates therespective correlated end-user events, contextual information associatedwith the end-user events, REST calls and return responses intorespective event vectors for each client computing device 20. The RESTcalls and return responses may be referred to as REST services. An eventvector may be represented as follows:Event_Vector=f(end-user event, contextual information, REST services)

The corresponding REST calls and return responses from the proxy 50include parameter information noise, and the processor 62 is configuredto remove the parameter information noise by applying one or morereducer functions. Parameter information noise refers to the parameterinformation contained within the REST services which can lead to a widevariety of vectors for end-users.

For instance, when end-users log-in and go to particular customers, theparticular customers are different. One end-user goes to customer A andanother end-user goes to customer C, for example. Since both end-usersare going from the log-in page to a customer, it does not matter whichcustomer. The common activity being performed within the SalseForce isthe fact that end-users are going from the log-in page to a customer.Customer specifics is an example of parameter information noise.

One or more reducer functions are applied by the processor 62 totranslate each event vector into one or more reduced event vectors. Thereduced event vectors do not include the parameter information noise. Areduced event vector may be represented as follows:Reduced_Event_Vector=R(Event_Vector)where R is the reducer function. Multiple reducer functions may bedefined depending on the type of parameter information noise collectedby the embedded browser 24.

A reducer function is based on skip-grams. Skip grams may also bereferred to as k-skip-n-grams where it tries multiple variants (many) ofskipping n events over a sequence of k events until it finds a betterclustering of reduced vectors. Skip-grams reported for a certain skipdistance k allow a total of k or less skips to construct the n-gram. Assuch, “4-skip-n-gram” results include 4 skips, 3 skips, 2 skips, 1 skip,and 0 skips.

All reduced end-user events are placed in sequence to collect end-userbehavior for each client computing device 20. An activity may be definedby a common sequence of reduced event vectors across the end-users.Detecting activities is implemented by extracting n-sets of reducedevent vectors from the sequence of events. An n-set of reduced eventvectors is a subset of subsequent reduced event vectors in sequence,where n>1. An activity within the application corresponds to theselected n-set of event vectors in sequence.

Since it is difficult to detect the beginning and end of an activity,the value of n is selected using a sliding window. The value of n is tobe selected so that the n-set of reduced event vectors in sequencecorresponds to a pattern that includes a beginning and an end of thecorresponding activity.

The processor 62 incrementally varies n for the n-set of event vectors,and generates an activity vector for each value of n. An activity vectormay also be referred to as a composite activity vector since the reducedevent vectors in the n-set of event vectors in sequence are addedtogether for each client computing device 20. The processor 62 examineseach composite activity vector for a pattern. In this example, theactivity is going from a log-in page to a customer.

As an example, n may start at 2. The processor 62 generates a compositeactivity vector with 2 of the reduced event vectors that are in sequenceand examines the composite activity vector for a common pattern. If acommon pattern is not detected, then the value of n is incremented to 3by the processor 62. The processor 62 now generates a composite activityvector by adding the next reduced event vector in sequence (i.e., n=3)and examines the composite activity vector for a common pattern. Thisprocess is repeated until a common pattern is detected, i.e., end-usersgoing from a log-in page to a customer in SalesForce.

By having large volumes of these sequences the processor 62 is able tofind common end sets and make a determination that this is apparently apattern. There will be a series of reduced event vectors added togetherrepresenting end-users going from a log-in page to a customer. At somepoint there is no more benefit to have n too large since commonality islost.

The processor 62 keeps a history of the composite activity vectors forall of the subsets of the sliding window. Each of the composite activityvectors for each client computing device 20 are placed in vector space.Cosine calculations are performed by the processor 62 to detect similaractivities across all users.

The similar activities are associated with a quality indicator toidentify anomalies within the application 42 for corrective action to betaken. The quality indicator may be based on one or more statisticsassociated with the end-users accessing the application 42. Thestatistical data tends to be more accurate for larger sets of clientcomputing devices 20.

Statistical data may include calculation of standard deviations for theend-users operating the client computing devices 20 for performingparticular activities within the application 42. Statistical data mayalso include calculation on the frequency and speed of performingparticular activities within the application 42.

An anomaly, for example, corresponds to an end-user taking too long toperform an activity as compared to the other end-users. In this case,corrective action includes recommending to the end-user which end-usersevents to use for more efficiently performing the activity. Thisrecommendation may be automatically generated and provided to theend-user by the analytics server 60. Alternatively, the recommendationis provided by an administrator of the analytics server 60.

Another anomaly, for example, corresponds to an end-user performing anactivity more frequently than the other end-users. For example, anend-user accesses a customer in SalseForce and then copies the retrievedcustomer information into a directory on their client computing device20. If this end-user is doing this for an unusually large number ofcustomers, as compared to the other end-users, then there could be asecurity concern. The administrator of the analytics server 60 willinvestigate to see if the end-user is stealing customer data.

Another anomaly, for example, may result in quality improvements to theapplication 42. For example, a large number of end-users access acustomer and then delete a customer. These are separate activitiesperformed by the end-users. It would be more efficient for theseend-users to perform this access and deletion as part of the sameactivity. In this case, a request would be made to the administrator ofthe web server 40 to include this feature in their application 42.

Referring now to the flowchart 200 in FIG. 4, and generally speaking, amethod for operating the computing system 10 will be discussed. From thestart (Block 202), the method includes operating client computingdevices 20 to access an application 42 at a server 40 based on end-userevents resulting in REST calls to the server 40 at Block 204. A proxy 50is operated at Block 206 to pass through the REST calls to the server 40and to return responses from the server 40, with the return responsescorresponding to activities being performed within the application 42.An analytics server 60 is operated at Block 208 to analyze sequences ofREST calls related to the end-user events to compare similar activitiesacross a large set of end-users accessing the same application 42. Themethod ends at Block 210.

Referring now to the flowchart 300 in FIG. 5, and generally speaking, amethod for operating the analytics server 60 within the computing system10 will be discussed. From the start (Block 302), the method includesreceiving the end-user events from each client computing device 20 atBlock 304, and receiving the REST calls and the return responses fromthe proxy 306 at Block 304. The end-user events are correlated with thecorresponding REST calls and return responses from the proxy 50 for eachclient computing device 20 at Block 308. The method further includestranslating respective correlated end-user events, REST calls and returnresponses into respective event vectors at Block 310. The respectiveevent vectors are processed at Block 312 to determine similarities amongthe plurality of client computing devices, with the similaritiescorresponding to similar activities. The similar activities areassociated with a quality indicator at Block 314 to identify anomalieswithin the application for corrective action to be taken. The methodends at Block 316.

Referring now to the flowchart 400 in FIG. 6, a more detailed flowchartfor processing the respective event vectors in the flowchart of FIG. 5will be discussed. From the start (Block 402), the processing comprisesplacing the respective event vectors in sequence for each clientcomputing device 20 at Block 404, and selecting an n-set of eventvectors in sequence from the sequence of event vectors for each clientcomputing device 20 at Block 406, where n>1. An activity within theapplication 42 corresponds to the selected n-set of event vectors insequence. A composite activity vector is generated at Block 408 byadding together the event vectors in the n-set of event vectors insequence for each client computing device 20. The composite activityvectors for each client computing device 20 are placed in vector spaceat Block 410. The similarities among the client computing devices 20 aredetermined at Block 412 based on similarities between the compositeactivity vectors among the plurality of client computing devices 20. Theprocessing ends at Block 414.

Another aspect is directed to a non-transitory computer readable mediumfor operating an analytics server 60 within a computing system 10comprising a web server 40 to provide an application 42; a plurality ofclient computing devices 20, with each client computing device 20 beingoperated by an end-user to access the application 42 based on end-userevents resulting in representational state transfer (REST) calls to theweb server 40; and a proxy 50 between the web server 40 and theplurality of client computing devices 20, and configured to pass throughthe REST calls to the web server 40 and to return responses from the webserver 40, with the return responses corresponding to activities beingperformed within the application 42.

The non-transitory computer readable medium has a plurality of computerexecutable instructions for causing the analytics server 50 to receivingthe end-user events from each client computing device 20, and receivingthe REST calls and the return responses from the proxy 306. The end-userevents are correlated with the corresponding REST calls and returnresponses from the proxy 50 for each client computing device 20.Respective correlated end-user events, REST calls and return responsesare translated into respective event vectors. The respective eventvectors are processed to determine similarities among the plurality ofclient computing devices 20, with the similarities corresponding tosimilar activities. The similar activities are associated with a qualityindicator to identify anomalies within the application for correctiveaction to be taken.

Many modifications and other embodiments will come to the mind of oneskilled in the art having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it isunderstood that the disclosure is not to be limited to the specificembodiments disclosed, and that modifications and embodiments areintended to be included within the scope of the appended claims.

That which is claimed:
 1. A computing system comprising: a web server toprovide an application; a plurality of client computing devices, witheach client computing device being operated by an end-user to access theapplication based on end-user events resulting in representational statetransfer (REST) calls to said web server; a proxy between said webserver and said plurality of client computing devices, and configured topass through the REST calls to said web server and to return responsesfrom said web server, with the return responses corresponding toactivities being performed within the application; and an analyticsserver configured to receive the end-user events from each clientcomputing device, and to receive the REST calls and the return responsesfrom said proxy, with said analytics server comprising a processorconfigured to perform the following: correlate the end-user events withthe corresponding REST calls and return responses from said proxy foreach client computing device, translate respective correlated end-userevents, REST calls and return responses into respective event vectors,process the respective event vectors to determine similarities amongsaid plurality of client computing devices, with the similaritiescorresponding to similar activities, and associate the similaractivities with a quality indicator to identify anomalies within theapplication for corrective action to be taken.
 2. The computing systemaccording to claim 1 wherein processing the respective event vectorscomprises: place the respective event vectors in sequence for eachclient computing device; select an n-set of event vectors in sequencefrom the sequence of event vectors for each client computing device,where n>1, and with an activity within the application corresponding tothe selected n-set of event vectors in sequence; generate a compositeactivity vector by adding together the event vectors in the n-set ofevent vectors in sequence for each client computing device; and placethe composite activity vectors for each client computing device invector space; with the similarities among said plurality of clientcomputing devices being determined based on similarities between thecomposite activity vectors among said plurality of client computingdevices.
 3. The computing system according to claim 2 whereindetermining the similarities comprising calculating a cosine distancebetween the composite activity vectors.
 4. The computing systemaccording to claim 1 wherein each client computing device comprises anembedded browser to monitor the end-user events by capturing events atthe document object model (DOM) level within a web page.
 5. Thecomputing system according to claim 4 wherein each client computingdevice comprises a processor configured to cooperate with said embeddedbrowser to extract end-user event attributes from the DOM, with theend-user event attributes providing contextual information on thecorresponding end-user event.
 6. The computing system according to claim5 wherein said analytics server receives the contextual informationassociated with the end-user events, and wherein said processor isfurther configured to include the contextual information in therespective event vectors.
 7. The computing system according to claim 1wherein the corresponding REST calls and return responses from saidproxy include parameter information noise, and where said processor isfurther configured to remove the parameter information noise by applyingone or more reducer functions.
 8. The computing system according toclaim 7 wherein applying the one or more reducer functions comprisestranslating each event vector into one or more reduced event vectors,with the resulting reduced event vectors being used as the event vectorsby said processor.
 9. The computing system according to claim 1 whereinthe n-set of event vectors in sequence corresponds to a pattern thatincludes a beginning and an end of the corresponding activity.
 10. Thecomputing system according to claim 9 wherein said processor is furtherconfigured to perform the following: incrementally vary n for the n-setof event vectors; generate an activity vector for each value of n;examine each activity vector for a pattern; and select the ncorresponding to the activity vector having a pattern.
 11. The computingsystem according to claim 1 wherein the quality indicator is based onone or more statistics associated with the end-users accessing theapplication.
 12. The computing system according to claim 1 wherein thecorrective action comprises recommending to one or more end-users whichend-users events to use for performing a particular activity within theapplication.
 13. The computing system according to claim 1 wherein theapplication comprises a Software as a Service (SaaS) application.
 14. Amethod for operating an analytics server within a computing systemcomprising a web server to provide an application; a plurality of clientcomputing devices, with each client computing device being operated byan end-user to access the application based on end-user events resultingin representational state transfer (REST) calls to the web server; and aproxy between the web server and the plurality of client computingdevices, and configured to pass through the REST calls to the web serverand to return responses from the web server, with the return responsescorresponding to activities being performed within the application, themethod comprising: receiving the end-user events from each clientcomputing device; receiving the REST calls and the return responses fromthe proxy; correlating the end-user events with the corresponding RESTcalls and return responses from the proxy for each client computingdevice; translating respective correlated end-user events, REST callsand return responses into respective event vectors; processing therespective event vectors to determine similarities among the pluralityof client computing devices, with the similarities corresponding tosimilar activities; and associating the similar activities with aquality indicator to identify anomalies within the application forcorrective action to be taken.
 15. The method according to claim 14wherein processing the respective event vectors comprises: placing therespective event vectors in sequence for each client computing device;selecting an n-set of event vectors in sequence from the sequence ofevent vectors for each client computing device, where n>1, and with anactivity within the application corresponding to the selected n-set ofevent vectors in sequence; generating a composite activity vector byadding together the event vectors in the n-set of event vectors insequence for each client computing device; and placing the compositeactivity vectors for each client computing device in vector space; withthe similarities among the plurality of client computing devices beingdetermined based on similarities between the composite activity vectorsamong the plurality of client computing devices.
 16. The methodaccording to claim 14 wherein each client computing device comprises anembedded browser to monitor the end-user events by capturing events atthe document object model (DOM) level within a web page.
 17. The methodaccording to claim 16 wherein each client computing device comprises aprocessor configured to cooperate with the embedded browser to extractend-user event attributes from the DOM, with the end-user eventattributes providing contextual information on the correspondingend-user event.
 18. The method according to claim 14 wherein the qualityindicator is based on one or more statistics associated with theend-users accessing the application.
 19. The method according to claim14 wherein the corrective action comprises recommending to one or moreend-users which end-users events to use for performing a particularactivity within the application.
 20. A non-transitory computer readablemedium for operating an analytics server within a computing systemcomprising a web server to provide an application; a plurality of clientcomputing devices, with each client computing device being operated byan end-user to access the application based on end-user events resultingin representational state transfer (REST) calls to the web server; and aproxy between the web server and the plurality of client computingdevices, and configured to pass through the REST calls to the web serverand to return responses from the web server, with the return responsescorresponding to activities being performed within the application, andwith the non-transitory computer readable medium having a plurality ofcomputer executable instructions for causing the analytics server toperform steps comprising: receiving the end-user events from each clientcomputing device; receiving the REST calls and the return responses fromthe proxy; correlating the end-user events with the corresponding RESTcalls and return responses from the proxy for each client computingdevice; translating respective correlated end-user events, REST callsand return responses into respective event vectors; processing therespective event vectors to determine similarities among the pluralityof client computing devices, with the similarities corresponding tosimilar activities; and associating the similar activities with aquality indicator to identify anomalies within the application forcorrective action to be taken.